Prompting Machine-Learned Models Using Chains of Thought

ABSTRACT

Example embodiments of aspects of the present disclosure provide an example computer-implemented method for improved prompting of a machine-learned model. The example method can include obtaining an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response. The example method can include inputting, to a machine-learned model, the instructive sequence and an operative query, wherein the machine-learned model is configured to process the operative query with attention over the instructive sequence. The example method can include generating, using the machine-learned model and responsive to the operative query, an operative response.

RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalPatent Application No. 63/348,637, filed Jun. 3, 2022. U.S. ProvisionalPatent Application No. 63/348,637 is hereby incorporated by referenceherein in its entirety.

FIELD

The present disclosure relates generally to the control ofmachine-learned models. More particularly, the present disclosurerelates to constructing prompting inputs for machine-learned models.

BACKGROUND

Machine-learned models can provide various functionality. Such modelscan be trained to perform various tasks. Already-trained models can befurther instructed to perform particular tasks by providing inputs tothe model with rich context that prompts the model to behave in adesired fashion.

SUMMARY

Aspects and advantages of embodiments of the present disclosure will beset forth in part in the following description, or can be learned fromthe description, or can be learned through practice of the embodiments.

In one example aspect, example embodiments of the present disclosureprovide for an example computer-implemented method for improvedprompting of a machine-learned model. The example method includesobtaining, by a computing system including one or more processors, aninstructive sequence descriptive of an instructive query, an instructiveresponse, and an instructive trace of intermediate states from theinstructive query to the instructive response. The example methodincludes inputting, by the computing system and to a machine-learnedmodel, the instructive sequence and an operative query, wherein themachine-learned model is configured to process the operative query withattention over the instructive sequence. The example method includesgenerating, by the computing system, using the machine-learned model andresponsive to the operative query, an operative response.

In one example aspect, example embodiments of the present disclosureprovide for one or more example memory devices storing computer-readableinstructions for improved prompting of a machine-learned model, theinstructions executable to cause one or more processors to performexample operations. The example operations include obtaining aninstructive sequence descriptive of an instructive query, an instructiveresponse, and an instructive trace of intermediate states from theinstructive query to the instructive response. The example operationsinclude inputting, to a machine-learned model, the instructive sequenceand an operative query, wherein the machine-learned model is configuredto process the operative query with attention over the instructivesequence. The example operations include generating, using themachine-learned model, a plurality of operative responses. The exampleoperations include determining a consistency metric based on a sample ofthe plurality of operative responses. The example operations includedetermining an operative response based on the consistency metric.

In one example aspect, example embodiments of the present disclosureprovide for an example computing system for improved prompting of amachine-learned model. The example system includes one or moreprocessors and one or more memory devices storing computer-readableinstructions executable to cause the one or more processors to performexample operations. In the example system, the example operationsinclude obtaining an instructive sequence descriptive of an instructivequery, an instructive response, and an instructive trace of intermediatestates from the instructive query to the instructive response. In theexample system, the example operations include inputting, to amachine-learned model, the instructive sequence and an operative query,wherein the machine-learned model is configured to process the operativequery with attention over the instructive sequence. In the examplesystem, the example operations include generating, using themachine-learned model, a plurality of operative responses. In theexample system, the example operations include determining a consistencymetric based on a sample of the plurality of operative responses. In theexample system, the example operations include determining an operativeresponse based on the consistency metric.

Other aspects of the present disclosure are directed to various systems,apparatuses, non-transitory computer-readable media, user interfaces,and electronic devices.

These and other features, aspects, and advantages of various embodimentsof the present disclosure will become better understood with referenceto the following description and appended claims. The accompanyingdrawings, which are incorporated in and constitute a part of thisspecification, illustrate example embodiments of the present disclosureand, together with the description, serve to explain the relatedprinciples.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill inthe art is set forth in the specification, which makes reference to theappended figures, in which:

FIG. 1 depicts a block diagram of an example input data structure andcorresponding example out for chain of thought prompting according toexample aspects of some embodiments of the present disclosure;

FIG. 2 depicts a block diagram of an example input data structure andcorresponding example out for chain of thought prompting according toexample aspects of some embodiments of the present disclosure;

FIG. 3 depicts a block diagram of an example input data structure andcorresponding example out for chain of thought prompting according toexample aspects of some embodiments of the present disclosure;

FIG. 4 depicts a block diagram of an example input data structure andcorresponding example out for chain of thought prompting according toexample aspects of some embodiments of the present disclosure;

FIG. 5 depicts a block diagram of an example input data structure andcorresponding example out for recursive prompting according to exampleaspects of some embodiments of the present disclosure;

FIG. 6 depicts example results for benchmark comparisons for chain ofthought prompting according to example aspects of some embodiments ofthe present disclosure;

FIG. 7 depicts example results for benchmark comparisons for chain ofthought prompting according to example aspects of some embodiments ofthe present disclosure;

FIG. 8 depicts example results for benchmark comparisons for chain ofthought prompting according to example aspects of some embodiments ofthe present disclosure;

FIG. 9 depicts example results for benchmark comparisons for chain ofthought prompting according to example aspects of some embodiments ofthe present disclosure;

FIG. 10A depicts a block diagram of an example computing system thatperforms chain of thought prompting according to example aspects of someembodiments of the present disclosure;

FIG. 10B depicts a block diagram of an example computing device thatperforms chain of thought prompting according to example aspects of someembodiments of the present disclosure;

FIG. 10C depicts a block diagram of an example computing device thatperforms chain of thought prompting according to example aspects of someembodiments of the present disclosure; and

FIG. 11 depicts a flow chart diagram of an example method to performchain of thought prompting according to example aspects of someembodiments of the present disclosure.

Reference numerals that are repeated across plural figures are intendedto identify the same features in various implementations.

DETAILED DESCRIPTION Overview

Generally, the present disclosure is directed to improved techniques forprompting machine-learned models to perform various tasks. Exampleembodiments of the present disclosure relate to prompting amachine-learned model using a “chain of thought” that traces thereasoning used to generate an output responsive to a given input. Forexample, a machine-learned model can be trained (e.g., in pre-training,fine tuning, etc.) to learn relationships between inputs. For instance,a machine-learned model can be trained to learn relationships betweenterms in an input query. Prompting a machine-learned model can includeproviding an instructive input query and an instructive output responsebefore an operative query of interest. By also providing an instructivetrace explaining the sequence of reasoning steps or logical statesbetween the instructive input query and the instructive output response,example prompts according to aspects of the present disclosure canbetter leverage the network of learned associations to communicate moreinstructive context with a given prompt.

For example, traditional model input structures can be suitable for sometasks. For instance, scaling up the size of language models has led toimprovements in performance and sample efficiency. For instance,language models at the scale of 100B or more parameters have achievedstrong performance on natural language processing tasks such assentiment analysis and topic classification, even in few-shot andzero-shot settings.

However, on other tasks, even large models can struggle usingtraditional input and control techniques. For instance, usingtraditional input and control techniques, even large language models canstruggle with tasks that involve slow and deliberate thinking (e.g.,“system-2 tasks,” tasks with multiple steps, etc.), and includeslogical, mathematical, and commonsense reasoning tasks, among others.This difficulty can arise even when models are scaled into the hundredsof billions of parameters. For example, a pre-trained GPT-3 model canstruggle to perform few-shot addition on numbers with greater than threedigits. Similarly, existing large-scale language model implementationscan struggle to predict the result of executing Python code, even codewhich is a solution to a programming task the model is generally able tosolve. And standard recurrent and graph neural network implementationscan fail to systematically generalize when predicting the output ofsimple programs with loops.

Advantageously, example techniques of the present disclosure can enablemachine-learned models to decompose a posed query or problem intointermediate steps that are solved individually. In some examples, thistechnique enables the model to resolve the intermediate steps instead ofsolving an entire multi-hop problem in a single forward pass, provingcapacity to focus the model's processing power on more challengingintermediate steps instead of spreading the compute resources thin overall steps at once. Examples of this technique enable the model toresolve the intermediate steps in concert with resolution of the desiredoutput value, leveraging the richer context of the reasoning trace toguide and refine the desired output value.

For example, in some embodiments, machine-learned models can beinstructed to generate such chains of thought as intermediate traces.For example, single-shot or few-shot prompting using a number ofinstructive examples can provide a pattern that the model can understandand follow. In some examples, including an instructive trace with theinstructive examples enables the model to generate its own trace whenprocessing a query.

In some embodiments, a machine-learned model can output a single queryresponse and trace thereof. In some embodiments, a machine-learned modelcan output a plurality of responses (and corresponding traces). Theplurality of responses can be leveraged to determine a consistencymetric. For instance, a consistency metric can be evaluated across asampling of diverse traces (e.g., representing diverse approaches toresolving the query) and corresponding responses. For example, a set ofoutputs with diverse reasoning strategies can be polled to obtain amajority or plurality “vote” on the ultimate answer. In this manner, themodel output can self-corroborate its “rationale” to improve therobustness of model output and improve accuracy of the ultimate answers.Compared to some prior decoding methods, a self-consistency techniqueaccording to the present disclosure can avoid the repetitiveness thatcan affect greedy sampling, while mitigating the stochasticity of asingle random generation. Compared to prior generate-then re-rankapproaches, self-consistency can avoid using a specially-trainedre-ranker and can have a faster runtime (e.g., given the same number ofdecodes).

In some embodiments, a chain of thought can span multiple queriesprocessed by the machine-learned model. For instance, a target query mayinclude a complex or multi-part question. The target query can be brokendown or reduced into one or more query components (e.g., using promptingor other methods, using the same or a different model, etc.). The querycomponents can then be recursively processed by the model. For instance,a first query component can be processed in view of an initialinstructive sequence (e.g., a chain-of-thought prompt as describedherein, etc.). In some embodiments, each successive query component canbe processed in view of prior query components and responses thereto.For instance, in this manner, the machine-learned model canself-construct an updated instructive sequence with each recursion toleverage its own prior work to build toward an ultimate response to thetarget query.

Example embodiments of input data structures according to aspects of thepresent disclosure can provide for a number of technical effects andbenefits. In some embodiments, causing a machine-learned model togenerate a chain of thought according to aspects of the presentdisclosure can provide an interpretable window into the behavior of themodel, suggesting how it might have arrived at a particular answer andproviding opportunities to debug where the reasoning path went wrong.Input data structures configured according to example embodiments of thepresent disclosure can unlock previously unrealized capabilities tounderstand, audit, debug, and improve the functionality of computingdevices executing machine-learned models.

In some embodiments, input data structures configured according toexample embodiments of the present disclosure can enable machine-learnedmodels to be used for cross-domain tasks. For instance, amachine-learned model trained on a textual corpus may contain weightswhich encode a number of semantic associations between concepts. Usingan input data structure configured according to the present disclosure,such a model can provide utility in resolving queries for any problemthat can be formulated in a textual expression, even if the model wasnot trained to perform such a problem type (e.g., mathematical problems,symbolic manipulation more generally, etc.). In this manner, forexample, the presently disclosed input data structures unlock the fullcomputational power of machine-learned models to solve new problemsoutside of a training domain.

In some embodiments, input data structures configured according toexample embodiments of the present disclosure can provide for animproved human-machine interface for inputting and processing queries.For instance, in the context of machine-learned language models, inputdata structures according to the present disclosure enable a user tocontrol the model to perform complex calculations or other reasoningtasks by inputting only simple instructive strings. In this manner, thetechnological power of complex machine-learned language models can bemade more accessible to non-technical users who may lack requisitetraining or other resources to, for example, fine-tune amultibillion-parameter model to perform a particular task. By improvingthe interface for such models, example embodiments of the presentdisclosure improve the capabilities of computing devices executing themodels in such implementations by providing for new pathways ofinteraction with the models.

In some embodiments, input data structures configured according toexample embodiments of the present disclosure can provide for decreasedusage of computing resources to adapt a model to a given task. Forinstance, traditional approaches to instructing a machine-learned modelto perform a given task include updating model parameter(s) based on anobjective evaluated over some training input. Such an update procedurecan be extremely resource intensive (e.g., computational resources,electrical resources, etc.) and may be cost-prohibitive (e.g., energycost, time cost, etc.). In contrast, input data structures according tothe present disclosure can provide for adaptation of large models (e.g.,billions of parameters, trillions of parameters, etc.) withoutnecessarily requiring additional training. For instance, input datastructures according to the present disclosure can provide forimprovements in model performance with just one or more instructiveexamples and instructive traces.

With reference now to the Figures, example embodiments of the presentdisclosure will be discussed in further detail.

FIG. 1 depicts an example configuration of prompting a machine-learnedmodel 100 according to aspects of the present disclosure. An input datastructure 102 can include an instructive sequence 104 that contains aninstructive query 106, an instructive trace 108, and an instructiveresponse 110. Multiple different instructive sequences 104 can beprovided in the input data structure 102. The input data structure 102can also include an operative query 112. The instructive query 106,instructive trace 108, instructive response 110, and operative query 112can contain embedded values. For instance, an embedded value can includea tokenized representation of an input string (e.g., text string,symbolic string, etc.). In some embodiments, an embedded value caninclude a tokenized representation of other data (e.g., image data,etc.).

In some embodiments, the machine-learned model 100 includes a neuralnetwork trained to understand and interpret inputs to generate anoutput. For instance, in some embodiments, the machine-learned model 100includes a neural network trained to understand and interpret text orother symbolic inputs to extract semantic meaning therefrom, includingto respond to instructions provided in such inputs. In some embodiments,the machine-learned model 100 includes a neural network trained tounderstand and interpret images or other data inputs more generally toextract meaning therefrom, including to respond to instructions providedin such inputs.

In general, the techniques and input data structures of the presentdisclosure can be implemented using and adapted for a variety of modelarchitectures. In some embodiments, the machine-learned model 100 isconfigured to attend over the instructive sequence 204 when processingthe operative query 112. For instance, in some embodiments, themachine-learned model 100 can include one or more transformerarchitectures (e.g., encoder only, decoder only, encoder and decoder,etc.).

In some embodiments, the instructive query 104 can present substantiallyany type of problem, question, or task to be performed. For instance,the instructive query 104 can include substantially any problem capableof being explained, reasoned, or otherwise expressed with symbols,images, language, etc. For example, the instructive query 104 caninclude mathematical queries, logic queries, knowledge queries,generative queries, summary queries, analytics queries, retrievalqueries, image processing queries, etc.

In some embodiments, the instructive trace 108 can include one or moreintermediate states from the instructive query 106 to the instructiveresponse 110. For example, intermediate states can include intermediatevalues associated with component subtasks, declarations of knownsdetermined (explicitly or implicitly) from the instructive query,logical steps to progress from a problem to a solution, a log ofsubtasks performed to generate the instructive response 110, etc.

The instructive response 110 can include the fulfillment of theinstructive query 106. For instance, in some embodiments of amathematical instructive query 106, the instructive response 110 caninclude a numerical solution, an analytical or symbolic solution, etc.In some embodiments, for a knowledge instructive query 106, theinstructive response 110 can include returning the requested knowledge,etc.

In some embodiments, the operative query 112 can be of a similar type ofquery to the instructive query 106. In some embodiments, the operativequery 112 can be of a different type of query to the instructive query106 (e.g., when multiple instructive sequences 104 are provided).

In some embodiments, the instructive query 106 and operative query 112can contain input flag(s) and output flag(s). For instance, theinstructive query 106 can contain an input flag indicating a query startposition and an output flag indicating a portion to be generated by themodel 100 (e.g., a subsequent portion of the instructive sequence 104).

Based on the input data structure 102, the machine-learned model 100 cangenerate an output 120. In some embodiments, the output 120 can containan operative trace 122 and an operative response 124. Generally, theoperative response 124 can include a fulfillment of the operative query112 (e.g., including an expression of an inability to fulfill the query,etc.). In some embodiments, the operative trace 112 can be generatedbased on a pattern set by one or more instructive traces in the inputdata structure 102. In some embodiments, the operative response 124 canbe generated to relate to the operative trace 122 and the operativequery 112 based on a pattern set by the instructive sequence(s) 104.

FIG. 2 illustrates one example implementation of an input data structure202 according to aspects of the present disclosure. Instructive sequence204 can include an instructive query 206 which embeds, represents, orotherwise is descriptive of a query corresponding to the string “Q:Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each canhas 3 tennis balls. How many tennis balls does he have now? A:” In theexample instructive query 206, “Q:” can correspond to an input flagindicating the start of an input query. In the example instructive query206, “A:” can correspond to an output flag indicating the start of aportion to be provided in response to the instructive query 206.

Instructive sequence 204 can include an instructive trace 208documenting intermediate states from the instructive query 206 to theinstructive response 210. For instance, although the direct answer tothe posed query is captured by the instructive response 210, “The answeris 11,” the instructive trace 208 can capture a series of intermediates(or the “chain of thought”) leading to the ultimate answer. Forinstance, a first intermediate state can include a declaration of aknown: “Roger started with 5 balls.” A second intermediate state caninclude a statement of multiplication based on the query values: “2 cansof 3 tennis balls each is 6 tennis balls.” A third intermediate statecan include a summation step (e.g., optionally numeric, in naturallanguage, etc.): “5+6=11.”

Operative query 212 can include a query of the same type as at least oneinstructive query 206. For instance, operative query 212 can include amathematical word problem of a similar type as the instructive query206: “Q: John takes care of 10 dogs. Each dog takes 0.5 hours a day towalk and take care of their business. How many hours a week does hespend taking care of dogs? A:”

The machine-learned model 100 can process the input data structure 202to generate output 220. The output 220 can include an operative trace222 and an operative response 224. For example, the operative trace 222can be generated to include one or more intermediate states ofreasoning/solution from the operative query 212 to the operativeresponse 224. For instance, a first intermediate state can include adeclarative statement of an explicit known, “John takes care of 10dogs.” A second intermediate state can include, for example, anotherdeclarative statement of an explicit known, “Each dog takes 0.5 hours aday to walk and take care of their business.” A third intermediate statecan include, for example, a statement of multiplication based on theexplicit knowns, “So that is 10×0.5=5 hours a day.” A fourthintermediate state can include, for example, a statement ofmultiplication based on an implicit known regarding the number of daysin a week, “5 hours a day×7 days a week=35 hours a week.” In thismanner, for example, the operative trace 222 can trace intermediatestate(s) from the operative query 212 to the operative response 224.

In some embodiments, the respective responses (e.g., instructiveresponse, operative response) can include the respective traces. Forinstance, in some examples the desired response is the trace. Forinstance, example embodiments can be implemented to obtain traces ofcomputer-executable script operation.

FIG. 3 depicts one example implementation of an input data structure 302in which an instructive sequence 304 contains an instructive query 306descriptive of a Python program (e.g., a tokenized representationthereof, etc.). In some examples, the instructive query 306 can includean input flag or an output flag. For instance, FIG. 3 depicts an inputflag “Consider the following Python function:” and an output flag “Whatis the execution trace? [BEGIN].” The instructive trace 308 can formpart of the instructive response 310, for example, because fulfillmentof the instructive query 304 corresponds to generation of the traceitself. The operative query 312 includes the input flag and output flagalong with a new Python program for tracing. Accordingly, the output 320generated by the machine-learned model 100 can include an operativetrace 322 forming part of the operative response 324.

In some embodiments, the machine-learned model 100 can directly generatean output for fulfilling the operative query. In some embodiments,fulfilling the operative query can include sampling a plurality ofoutputs to determine a response satisfying a consistency metric.

FIG. 4 provides an example illustration of an input data structure 402containing an instructive sequence 404 (including instructive query 406,instructive trace 408, and instructive response 410) and an operativequery 412. A machine-learned model 400 can be configured to output aplurality of outputs, including a plurality of operative tracescorresponding to a plurality of operative responses. A subset can besampled, for example, as sampled outputs 420, containing a first sampledoutput (operative trace 422-1, operative response 424-1), a secondsampled output (operative trace 422-2, operative response 424-2), and athird sampled output (operative trace 422-3, operative response 424-3).

In some embodiments, sampled outputs 420 can include a number of outputssampled from an output layer of a machine-learned model 400. In someembodiments, sampled outputs 420 can be sampled from a probabilitydistribution of the outputs (e.g., of a probability distribution overpairs of traces and responses). In some embodiments, samples areselected according to any suitable sampling scheme. In some embodiments,outputs are randomly sampled. In some embodiments, outputs can besampled based on a ranked probability (e.g., top-K outputs). In someembodiments, outputs can be sampled for diverse traces.

In some embodiments, a plurality or majority of diverse traces thatarrive at the same ultimate resolution can be indicative of a responseassociated with a higher confidence. Accordingly, in some embodiments, avote is taken over the sampled outputs (e.g., a plurality vote, amajority vote). For instance, a response selector 430 can determine thatthe ultimate answer of $18 is indicated in two out of the three sampledoutputs 420. In this manner, for example, a selected response 432 of $18can be obtained.

In some embodiments, evaluation of the consistency metric can beexpressed as applying a marginalization over the traces in theconditional probability P (response, trace query) of each output given aquery.

FIG. 5 depicts a block diagram of an example processing flow forperforming recursive prompting according to example aspects of thepresent disclosure. For instance, a machine-learned model pipeline caninclude one or more models 502, 504. The models 502 and 504 may be thesame or different. For instance, any one or both of model(s) 502, 504can be or contain models 100, 400, etc.

In a query breakdown stage 510, for example, a machine-learned model 502can reduce a complex problem into one or more component problems. Forinstance, in some embodiments, the model 502 can be prompted to performthe reduction with one or more instructive sequence(s) 512 (e.g., whichcan optionally contain instructive traces). In some embodiments, thetarget query 514 is input to the model 502. For instance, the targetquery 514 can include a scenario providing context for a question to beanswered (e.g., example question emphasized in bold in FIG. 5 ). Themodel 502 can generate one or more query components 516. In someembodiments, a query component can include a question that asks for partof an overall solution. In some embodiments, a query component caninclude a question that asks for a preliminary information componentthat can be used to obtain an overall solution. In some embodiments, aquery component can include a question that asks for a logicalcomplement, corollary, or other related component that mayadvantageously be easier to resolve.

In a query recursion stage 520, a machine-learned model 504 canrecursively process the query components 516 and optionally the initialtarget query 514. For instance, in some embodiments, the machine-learnedmodel 504 can be prompted with initial instructive sequences 522 toanswer the first query component. For instance, query component(s) 524can include the first query component from query components 516,optionally in combination with the scenario from the target query 514.In some embodiments, the initial instructive sequence(s) 522 can includeone or more instructive queries, instructive traces, and instructiveresponses according to example embodiments of the present disclosure. Insome embodiments, the query component(s) can correspond to an operativequery (e.g., as described with respect to FIGS. 1 to 4 ).

On one pass of query recursion 520, the model 504 can generate responsecomponent(s) 526 based on the input query component(s) and initialinstructive sequence(s) 522. For instance, the response component(s) 526can include an operative trace and an operative response.

To perform another pass of query recursion 520, a new instructivesequence can be composed from the body of prior knowledge about theproblem at hand, which can include new information generated by themodel 504. For instance, query component(s) 528 can incorporate querycomponent(s) 524 as well as the response component(s) 526. In thismanner, the prior work of the model 504 can effectively become aninstructive sequence including instructive queries, instructive traces,and instructive responses. Optionally, the initial instructive sequences522 can be retained for input together with the query component(s) 528.In this manner, for instance, the model 504 can process additional querycomponent(s) (e.g., the original target query, in bold) by leveragingits prior outputs to generate response component(s) 530.

Query recursion 520 can include, in some embodiments, a plurality ofiterations. In some embodiments, the iterative recursion can provide forself-constructed instructive sequences. In some embodiments, this canhelp the machine-learned model leverage its full power over individualcomponent queries while retaining the ability to build on its own priorwork. In some embodiments, this can improve generalization from easy todifficult problems (e.g., easy problems explained via instruction, withinference performed over more difficult problems).

For example, in some embodiments, the query breakdown 510 can providefor an ordered set of query component(s) 516. For instance, in someembodiments, the query component(s) 516 can include an ordering frombasic (or foundational) queries to complex (or follow-on) queries. Insome embodiments, the set of query components is naturally ordered byappending the task from the original target query to the set of querycomponent(s) 516 generated by the model. In this manner, for instance,the query component(s) 516 can include tractable component queries thatcan be resolved before tackling the task from the target query 514itself. FIG. 5 illustrates this example flow.

Example Results: Arithmetic Reasoning

Example results are presented herein for illustration purposes only. Itis to be understood that the various configurations presented in theexamples are selected for the purpose of illustration and comparison andare not to be interpreted as somehow limiting the scope of disclosure.

First, example results will be discussed with respect to themathematical word problem type query depicted in FIG. 2 . Such queriesprobe the ability of language models to perform arithmetic reasoningwhile focusing on problems solvable by elementary school children (ages6-10). Though such problems can be simple for humans, arithmeticreasoning is a task where language models can exhibit a flat scalingcurve (e.g., model performance increase can taper as model sizeincreases). Advantageously, providing a prompt comprising a fewinstructive traces according to the present disclosure can dramaticallyimprove performance on difficult math word problems for large languagemodels. When scaled to 540B parameters, chain of thought prompting canperform comparably with task-specific finetuned models on a variety oftasks, including surpassing the GSM8K benchmark introduced by Cobbe etal., Training Verifiers to Solve Math Word Problems, ARXIV.org (Oct. 27,2021). For arithmetic reasoning examples discussed herein, the followingdatasets are used:

-   -   (1) SingleOp (Roy et al., Reasoning about Quantities in Natural        Language, Transactions of the Association for Computational        Linguistics, 2015. doi: 10.1162/tacl_a_00118);    -   (2) SingleEq (Koncel-Kedziorski et al., MAWPS: A math word        problem repository, In Proceedings of the 2016 Conference of the        North American Chapter of the Association for Computational        Linguistics: Human Language Technologies, 2016. doi:        10.18653/v1/N16-1136);    -   (3) AddSub, (Hosseini et al., Learning to solve arithmetic word        problems with verb categorization, In Proceedings of the 2014        Conference on Empirical Methods in Natural Language Processing        (EMNLP), 2014. doi: 10.3115/v1/D14-1058);    -   (4) ASDiv (Miao et al., A diverse corpus for evaluating and        developing English math word problem solvers, In Proceedings of        the 58th Annual Meeting of the Association for Computational        Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.92);    -   (5) MultiArith, (Roy et al., Solving general arithmetic word        problems, In Proceedings of the 2015 Conference on Empirical        Methods in Natural Language Processing, 2015 doi:        10.18653/v1/D15-1202); and    -   (6) GSM8K (Cobbe et al., Training Verifiers to Solve Math Word        Problems, ARXIV.org (Oct. 27, 2021)).

As a baseline approach, standard few-shot prompting results are providedin which a language model is given in-context exemplars of input—outputpairs before outputting a prediction for a test-time example. Exemplarsare formatted as questions and answers before being fed into the model,and the model gives the answer directly.

For the example chain-of-thought prompting results, a set of eightinstructive sequences are used. This set is provided in Table 1.

The results are generated by using two collections of denseleft-to-right, decoder-only transformer language models. The firstcollection is based on LaMDA (Thoppilan et al., Lamda: Language modelsfor dialog applications, arXiv preprint arXiv:2201.08239), which hasmodels of 422M, 2B, 8B, 68B, and 137B parameters. The second collectionof models is PaLM (Chowdhery et al., PaLM: Scaling language modelingwith Pathways, arXiv preprint arXiv:2204.02311, 2022), which has sizesof 8B, 62B, and 535B parameters. In the present examples, outputs aresampled from the model using greedy decoding. For LaMDA, results arereported averaged over five random seeds, where each seed had adifferent randomly shuffled order of exemplars. LaMDA experiments didnot show large variance among different seeds, so PaLM results arereported using a single random seed.

Example results are presented in FIGS. 6 and 7 .

TABLE 1 Instructive Sequences for Arithmetic Reasoning Examples Q: Thereare 15 trees in the grove. Grove workers will plant trees in the grovetoday. After they are done, there will be 21 trees. How many trees didthe grove workers plant today? A: There are 15 trees originally. Thenthere were 21 trees after some more were planted. So there must havebeen 21 − 15 = 6. The answer is 6. Q: If there are 3 cars in the parkinglot and 2 more cars arrive, how many cars are in the parking lot? A:There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5. The answeris 5. Q: Leah had 32 chocolates and her sister had 42. If they ate 35,how many pieces do they have left in total? A: Originally, Leah had 32chocolates. Her sister had 42. So in total they had 32 + 42 = 74. Aftereating 35, they had 74 − 35 = 39. The answer is 39. Q: Jason had 20lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. Howmany lollipops did Jason give to Denny? A: Jason started with 20lollipops. Then he had 12 after giving some to Denny. So he gave Denny20 − 12 = 8. The answer is 8 Q: Shawn has five toys. For Christmas, hegot two toys each from his mom and dad. How many toys does he have now?A: Shawn started with 5 toys. If he got 2 toys each from his mom anddad, then that is 4 more toys. 5 + 4 = 9. The answer is 9. Q: There werenine computers in the server room. Five more computers were installedeach day, from monday to thursday. How many computers are now in theserver room? A: There were originally 9 computers. For each of 4 days, 5more computers were added. So 5 * 4 = 20 computers were added. 9 + 20 is29. The answer is 29. Q: Michael had 58 golf balls. On tuesday, he lost23 golf balls. On wednesday, he lost 2 more. How many golf balls did hehave at the end of wednesday? A: Michael started with 58 golf balls.After losing 23 on tuesday, he had 58 − 23 = 35. After losing 2 more, hehad 35 − 2 = 33 golf balls. The answer is 33. Q: Olivia has $23. Shebought five bagels for $3 each. How much money does she have left? A:Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 × 3 = 15dollars. So she has 23 − 15 dollars left. 23 − 15 is 8. The answer is 8.

Example Results: Symbolic Reasoning

Second, example results are presented for performing symbolic reasoningtasks. Although the symbolic reasoning tasks discussed here aregenerally simple for humans, machine-learned models can typicallyexhibit a flat scaling curve for such tasks. In some examples shownhere, solving intermediate steps of a symbolic reasoning task accordingto aspects of the present disclosure using chain of thought promptingallows models to perform tasks that are not solvable with standardprompting alone.

Three tasks are presented herein for the sake of illustration ofsymbolic manipulation functions: Last letter concatenation (toconcatenate the last letters of words in randomly concatenated namesfrom the top one-thousand first and last names from name census data);Reverse list (to reverse the order of a list of randomly sampled namesof everyday objects); and Coin flip (to answer whether a coin is stillheads up after people either flip or do not flip the coin).

For each task a test set is split into an in-domain test set for whichexamples had the same number of steps as the training/few-shotexemplars, as well as two out-of-domain (OOD) test sets, for whichevaluation examples had more steps than those in the exemplars. For lastletter concatenation, the model only sees exemplars of names with twowords, and then performs last letter concatenation on names with threeand four words. The same is done for the number of items in the reverselist task (in-domain=5, OOD={6, 7}) and the number of potential flips inthe coin flip task (in-domain=2, OOD={3, 4}).

Example results are given in FIG. 8 .

TABLE 2 Few-shot exemplars for full chain of thought prompt for the lastletter concatenation task. Q: Take the last letters of the words in“Elon Musk” and concatenate them. A: The last letter of “Elon” is “n”.The last letter of “Musk” is “k”. Concatenating them is “nk”. The answeris nk. Q: Take the last letters of the words in “Larry Page” andconcatenate them. A: The last letter of “Larry” is “y”. The last letterof “Page” is “e”. Concatenating them is “ye”. The answer is ye. Q: Takethe last letters of the words in “Sergey Brin” and concatenate them. A:The last letter of “Sergey” is “y”. The last letter of “Brin” is “n”.Concatenating them is “yn”. The answer is yn. Q: Take the last lettersof the words in “Bill Gates” and concatenate them. A: The last letter of“Bill” is “l”. The last letter of “Gates” is “s”. Concatenating them is“ls”. The answer is ls.

TABLE 3 Few-shot exemplars for chain of thought prompt for the reverselist task. Q: Reverse the sequence “cigar, umbrella, key, gum, alarm”.A: First is cigar. Second is umbrella. Third is key. Fourth is gum.Fifth is alarm. Now to reverse, change the order to: Fifth is alarm.Fourth is gum. Third is key. Second is umbrella. First is cigar. So theanswer is “alarm, gum, key, umbrella, cigar”. Q: Reverse the sequence“player, passport, umbrella, bottle, watch”. A: First is player. Secondis passport. Third is umbrella. Fourth is bottle. Fifth is watch. Now toreverse, change the order to: Fifth is watch. Fourth is bottle. Third isumbrella. Second is passport. First is player. So the answer is “watch,bottle, umbrella, passport, player”. Q: Reverse the sequence “coin,postcard, case, pen, wallet”. A: First is coin. Second is postcard.Third is case. Fourth is pen. Fifth is wallet. Now to reverse, changethe order to: Fifth is wallet. Fourth is pen. Third is case. Second ispostcard. First is coin. So the answer is “wallet, pen, case, postcard,coin”. Q: Reverse the sequence “laptop, lipstick, pen, bin, clock”. A:First is laptop. Second is lipstick. Third is pen. Fourth is bin. Fifthis clock. Now to reverse, change the order to: Fifth is clock. Fourth isbin. Third is pen. Second is lipstick. First is laptop. So the answer is“clock, bin, pen, lipstick, laptop”. Q: Reverse the sequence “key, pen,screen, file, cigar”. A: First is key. Second is pen. Third is screen.Fourth is file. Fifth is cigar. Now to reverse, change the order to:Fifth is cigar. Fourth is file. Third is screen. Second is pen. First iskey. So the answer is “cigar, file, screen, pen, key” Q: Reverse thesequence “card, stamp, book, water, glasses”. A: First is card. Secondis stamp. Third is book. Fourth is water. Fifth is glasses. Now toreverse, change the order to: Fifth is glasses. Fourth is water. Thirdis book. Second is stamp. First is card. The answer is 29. Q: Reversethe sequence “clock, coin, bottle, head, postcard”. A: First is clock.Second is coin. Third is bottle. Fourth is head. Fifth is postcard. Nowto reverse, change the order to: Fifth is postcard. Fourth is head.Third is bottle. Second is coin. First is clock. So the answer is“postcard, head, bottle, coin, clock”. Q: Reverse the sequence “battery,glasses, lighter, water, scissors”. A: First is battery. Second isglasses. Third is lighter. Fourth is water. Fifth is scissors. Now toreverse, change the order to: Fifth is scissors. Fourth is water. Thirdis lighter. Second is glasses. First is battery. So the answer is“scissors, water, lighter, glasses, battery”.

TABLE 4 Few-shot exemplars for full chain of thought prompt for thecoinflip task. Q: A coin is heads up. Ka flips the coin. Sherrie flipsthe coin. Is the coin still heads up? A: The coin was flipped by Ka andSherrie. So the coin was flipped 2 times, which is an even number. Thecoin started heads up, so after an even number of flips, it will stillbe heads up. So the answer is yes. Q: A coin is heads up. Jamey flipsthe coin. Teressa flips the coin. Is the coin still heads up? A: Thecoin was flipped by Jamey and Teressa. So the coin was flipped 2 times,which is an even number. The coin started heads up, so after an evennumber of flips, it will still be heads up. So the answer is yes. Q: Acoin is heads up. Maybelle flips the coin. Shalonda does not flip thecoin. Is the coin still heads up? A: The coin was flipped by Maybelle.So the coin was flipped 1 time, which is an odd number. The coin startedheads up, so after an odd number of flips, it will be tails up. So theanswer is no. Q: A coin is heads up. Millicent does not flip the coin.Conception flips the coin. Is the coin still heads up? A: The coin wasflipped by Conception. So the coin was flipped 1 time, which is an oddnumber. The coin started heads up, so after an odd number of flips, itwill be tails up. So the answer is no. Q: A coin is heads up. Sal flipsthe coin. Raymond does not flip the coin. Is the coin still heads up? A:The coin was flipped by Sal. So the coin was flipped 1 time, which is anodd number. The coin started heads up, so after an odd number of flips,it will be tails up. So the answer is no. Q: A coin is heads up.Conception flips the coin. Kristian does not flip the coin. Is the coinstill heads up? A: The coin was flipped by Conception. So the coin wasflipped 1 time, which is an odd number. The coin started heads up, soafter an odd number of flips, it will be tails up. So the answer is no.Q: A coin is heads up. Inga does not flip the coin. Elanor does not flipthe coin. Is the coin still heads up? A: The coin was flipped by no one.So the coin was flipped 0 times. The coin started heads up, and it wasnot flipped, so it is still heads up. So the answer is yes. Q: A coin isheads up. Ryan flips the coin. Shaunda flips the coin. Is the coin stillheads up? A: The coin was flipped by Ryan and Shaunda. So the coin wasflipped 2 times, which is an even number. The coin started heads up, soafter an even number of flips, it will still be heads up. So the answeris yes.

Example Results: “Common Sense” Reasoning

Third, example results are presented for tasks of reasoning aboutphysical and human interactions under the presumption of generalbackground knowledge. Four benchmark datasets are selected for theexample results:

(1) CommonsenseQA (Talmor et al., CommonsenseQA: A question answeringchallenge targeting commonsense knowledge, In Proceedings of the 2019Conference of the North American Chapter of the Association forComputational Linguistics: Human Chain of Thought Prompting ElicitsReasoning in Large Language Models Language Technologies, Volume 1 (Longand Short Papers), 2019. doi: 10.18653/v1/N19-1421) includes commonsensereasoning questions about the world involving complex semantics thatoften require prior knowledge;

-   -   (2) StrategyQA (Geva et al., Did aristotle use a laptop? A        question answering benchmark with implicit reasoning strategies,        Transactions of the Association for Computational        Linguistics, 2021. doi: 10.1162/tacl_a_00370) includes inference        of a multi-hop strategy to answer questions;    -   (3) Date Understanding, which involves inferring a date from a        given context; and    -   (4) Sports Understanding, which involves determining whether a        sentence relating to sports is plausible or implausible;    -   with (3) and (4) from (BIG-bench collaboration, Beyond the        imitation game: Measuring and extrapolating the capabilities of        language models, In preparation, 2021,        https://github.com/google/BIG-bench).

Example results are given in FIG. 9 .

TABLE 5 Few-shot exemplars for full chain of thought prompt forCommonsenseQA. Q: What do people use to absorb extra ink from a fountainpen? Answer Choices: (a) shirt pocket (b) calligrapher's hand (c)inkwell (d) desk drawer (e) blotter A: The answer must be an item thatcan absorb ink. Of the above choices, only blotters are used to absorbink. So the answer is (e). Q: What home entertainment equipment requirescable? Answer Choices: (a) radio shack (b) substation (c) television (d)cabinet A: The answer must require cable. Of the above choices, onlytelevision requires cable. So the answer is (c). Q: The fox walked fromthe city into the forest, what was it looking for? Answer Choices: (a)pretty flowers (b) hen house (c) natural habitat (d) storybook A: Theanswer must be something in the forest. Of the above choices, onlynatural habitat is in the forest. So the answer is (b). Q: Sammy wantedto go to where the people were. Where might he go? Answer Choices: (a)populated areas (b) race track (c) desert (d) apartment (e) roadblock A:The answer must be a place with a lot of people. Of the above choices,only populated areas have a lot of people. So the answer is (a). Q:Where do you put your grapes just before checking out? Answer Choices:(a) mouth (b) grocery cart (c) super market (d) fruit basket (e) fruitmarket A: The answer should be the place where grocery items are placedbefore checking out. Of the above choices, grocery cart makes the mostsense for holding grocery items. So the answer is (b). Q: Google Mapsand other highway and street GPS services have replaced what? AnswerChoices: (a) united states (b) mexico (c) countryside (d) atlas A: Theanswer must be something that used to do what Google Maps and GPSservices do, which is to give directions. Of the above choices, onlyatlases are used to give directions. So the answer is (d). Q: Beforegetting a divorce, what did the wife feel who was doing all the work?Answer Choices: (a) harder (b) anguish (c) bitterness (d) tears (e)sadness A: The answer should be the feeling of someone getting divorcedwho was doing all the work. Of the above choices, the closest feeling isbitterness. So the answer is (c).

TABLE 6 Few-shot exemplars for full chain of thought prompt forStrategyQA. Q: Do hamsters provide food for any animals? A: Hamsters areprey animals. Prey are food for predators. Thus, hamsters provide foodfor some animals. So the answer is yes. Q: Could Brooke Shields succeedat University of Pennsylvania? A: Brooke Shields went to PrincetonUniversity. Princeton University is about as academically rigorous asthe University of Pennsylvania. Thus, Brooke Shields could also succeedat the University of Pennsylvania. So the answer is yes. Q: Yes or no:Hydrogen's atomic number squared exceeds number of Spice Girls? A:Hydrogen has an atomic number of 1. 1 squared is 1. There are 5 SpiceGirls. Thus, Hydrogen's atomic number squared is less than 5. So theanswer is no. Q: Yes or no: Is it common to see frost during somecollege commencements? A: College commencement ceremonies can happen inDecember, May, and June. December is in the winter, so there can befrost. Thus, there could be frost at some commencements. So the answeris yes. Q: Yes or no: Could a llama birth twice during War in Vietnam(1945-46)? A: The War in Vietnam was 6 months. The gestation period fora llama is 11 months, which is more than 6 months. Thus, a llama couldnot give birth twice during the War in Vietnam. So the answer is no. Q:Yes or no: Would a pear sink in water? A: The density of a pear is about0.6 g = cm3, which is less than water. Objects less dense than waterfloat. Thus, a pear would float. So the answer is no.

TABLE 7 Few-shot exemplars for full chain of thought prompt for DateUnderstanding. Q: 2015 is coming in 36 hours. What is the date one weekfrom today in MM/DD/YYYY? A: If 2015 is coming in 36 hours, then it iscoming in 2 days. 2 days before 01/01/2015 is 12/30/2014, so today is12/30/2014. So one week from today will be 01/05/2015. So the answer is01/05/2015. Q: The first day of 2019 is a Tuesday, and today is thefirst Monday of 2019. What is the date today in MM/DD/YYYY? A: If thefirst day of 2019 was Tuesday, then 01/01/2019 was a Tuesday. Today isthe first monday, would be six days later. So today is 01/07/2019. Sothe answer is 01/07/2019. Q: The concert was scheduled to be on06/01/1943, but was delayed by one day to today. What is the date 10days ago in MM/DD/YYYY? A: One day after 06/01/1943 is 06/02/1943, sotoday is 06/02/1943. 10 days before today is 05/23/1943. So the answeris 05/23/1943. Q: It is 4/19/1969 today. What is the date 24 hours laterin MM/DD/YYYY? A: Today is 04/19/1969. 24 hours later is one day aftertoday, which would be 04/20/1969. So the answer is 04/20/1969. Q: Janethought today is 3/11/2002, but today is in fact Mar 12, which is 1 daylater. What is the date 24 hours later in MM/DD/YYYY? A: Today is03/12/2002. So the date 24 hours later will be 03/13/2002. So the answeris 03/13/2002. Q: Jane was born on the last day of Feburary in 2001.Today is her 16-year-old birthday. What is the date yesterday inMM/DD/YYYY? A: The last day of February is the 28th, so Jane was born on02/28/2001. Today is her 16-year old birthday, so today is 02/28/2017.So yesterday was 02/27/2017. So the answer is 02/27/2017.

TABLE 8 Few-shot exemplars for full chain of thought prompt for SportsUnderstanding. Q: Is the following sentence plausible? “Kyle Palmieriwas called for slashing.” A: Kyle Palmieri is a hockey player. Beingcalled for slashing is part of hockey. So the answer is yes. Q: Is thefollowing sentence plausible? “Joao Moutinho caught the screen pass inthe NFC championship.” A: Joao Moutinho is a soccer player. The NFCchampionship is part of American football, not soccer. So the answer isno. Q: Is the following sentence plausible? “Carson Wentz set the pickand roll.” A: Carson Wentz is an American football player. Pick and rollis part of basketball, not football. So the answer is no. Q: Is thefollowing sentence plausible? “Jonas Valanciunas beat the buzzer.” A:Jonas Valanciunas is a basketball player. Beating the buzzer is part ofbasketball. So the answer is yes. Q: Is the following sentenceplausible? “Jamel Murray was perfect from the line.” A: Jamal Murray isa basketball player. Being perfect from the line is part of basketball.So the answer is yes. Q: Is the following sentence plausible? “SamDarnold passed the puck.” A: Sam Darnold is a American football player.Passing the puck is part of hockey, not American football. So the answeris no. Q: Is the following sentence plausible? “Draymond Green threw atouchdown.” A: Draymond Green is an basketball player. Throwing atouchdown is part of football, not basketball. So the answer is no. Q:Is the following sentence plausible? “Malcolm Brogdon banked the shotin.” A: Malcolm Brogdon is a basketball player. Banking the shot in ispart of basketball. So the answer is yes.

Example Results: Self-Consistency

Example results for an example self-consistency technique according tothe present disclosure is provided over the following reasoningbenchmarks:

-   -   (1) Arithmetic reasoning: GSM8K, AddSub, MultiArith, and ASDiv        from above, as well as AQUA-RAT (Ling et al., Program induction        by rationale generation: Learning to solve and explain algebraic        word problems, In Proceedings of the 55th Annual Meeting of the        Association for Computational Linguistics (Volume 1: Long        Papers), 2017. doi:10.18653/v1/P17-1015) and SVAMP (Patel et        al., Are NLP models really able to solve simple math word        problems?, In Proceedings of the 2021 Conference of the North        American Chapter of the Association for Computational        Linguistics: Human Language Technologies, pp. 2080-2094).    -   (2) Commonsense reasoning: CommonsenseQA and StrategyQA (Geva et        al., 2021) for open-domain question-answering with implicit        multi-hop reasoning, and the AI2 Reasoning Challenge (ARC)        (Clark et al., Think you have solved question answering? Try        arc, the AI2 reasoning challenge, ArXiv, abs/1803.05457, 2018.).

Example self-consistency techniques were used to obtain results over thefollowing dense left-to-right, decoder-only transformer language modelswith varying scales:

-   -   (1) LaMDA-PT from above with 137-billion parameters, pretrained        on a mixture of web documents, dialog data and Wikipedia; and    -   (2) PaLM from above with 540-billion parameters, pretrained on a        high quality corpus of 780 billion tokens with filtered        webpages, books, Wikipedia, news articles, source code, and        social media conversations.

For the following example results, the same set of prompts presentedabove are used. Sampling scheme.

To sample diverse reasoning paths, for LaMDA-137B temperature samplingwas used with T=0.5 and truncated at the top-k (k=40) tokens with thehighest probability, and for PaLM-540B T=0.7, k=40. Example techniquesof self-consistency according to the present disclosure can be generallyrobust to sampling strategies and parameters. For sampled results, theresults are averaged over 10 runs, where 40 outputs are sampledindependently from the decoder in each run. Greedy decoding a singlechain of thought (e.g., as in previous examples) is provided forcomparison.

State-of-the-art results can be obtained on almost all tasks: despitethe fact that self-consistency is unsupervised and task-agnostic, theseresults compare favorably to more costly existing approaches thatrequire task-specific training, or fine-tuning with thousands ofexamples (e.g., on GSM8K). Example results are provided for arithmeticreasoning in Table 9. Example results on commonsense reasoning tasks aregiven in Table 10.

TABLE 9 Arithmetic reasoning results. Method AddSub MultiArith ASDivAQuA SVAMP GSM8K Previous SoTA 94.9 ^(a) 60.5^(a) 75.3^(b) 37.9^(c)57.4^(d) 35^(e)/57^(g) LaMDA Greedy decode (Single-path) 52.9 51.8 49.017.7 38.9 17.1 (137B) Self-Consistency (Multi-path) 63.5 (+10.6) 75.7(+23.9) 58.2 (+9.2) 26.8 (+9.1)  53.3 (+14.4) 27.7 (+10.6) PaLM Greedydecode (Single-path) 91.9 94.7 74.0 35.8 79.0 56.5 (540B)Self-Consistency (Multi-path) 93.7 (+1.8)  99.3 (+4.6)  81.9 (+7.9) 48.3(+12.5) 86.6 (+7.6)  74.4 (+17.9)

TABLE 10 Common Sense Reasoning Results. Method CommonsenseQA StrategyQAARC (Easy) ARC (Challenge) Previous SoTA 91.2^(a) 73.9^(b) 86.4^(c)75.0^(c) LaMDA Greedy decode (Single-path) 57.9 65.4 75.3 55.1 (137B)Self-Consistency (Multi-path) 63.1 (+5.2) 67.8 (+2.4) 79.3 (+4 0) 59.8(+4.7) PaLM Greedy decode (Single-path) 79.0 75.3 95.3 85.2 (540B)Self-Consistency (Multi-path) 80.7 (+1.7) 81.6 (+6.3) 96.4 (+1.1) 88.7(+3.5)

Example Results: Query Recursion

Example results are provided for the last-letter concatenation task. Inthis example task, the query includes a list of words, and the responseis the concatenation of the last letters of the words in the list. Forexample, “thinking, machine” outputs “ge” since the last letter of“thinking” is “g” and the last letter of “machine” is “e”. Theexperiment setup is as follows: (1) only two demonstration examples areprovided; and (2) the lists in training contain at most three words,while the lists for testing can be arbitrarily long. Although this taskis straightforward for humans, it is extremely challenging forstatistical machine learning methods. First, machine learning modelstrained with only two examples are not expected to generalize well.Second, the length-based train and test split requiresout-of-distribution generalization, which is highly non-trivial forstatistical learning.

The initial instructive sequences used for the Chain of Thought exampleand the Query Recursion example are provided in Table 10. Testing listswith lengths from 4 to 12 words were sampled from Wiktionary. For eachlength, 500 lists are constructed. Example results are given in Table11.

TABLE 10 Chain-of-thought and Query Recursion prompts for the examplelast letter concatenation task. Prompts for the naïve baseline aresimply input/output pairs. Chain of Thought Query Recursion Q: “think,machine” Q: “think, machine” A: The last letter of “think” is “k”. Thelast letter A: The last letter of “think” is “k”. The last of “machine”is “e”. Concatenating “k”, “e” letter of “machine” is “e”. Concatenatingleads to “ke”. So, “think, machine” outputs ”ke”. “k”, “e” leads to“ke”. So, “think, Q: “learning, reasoning, generalization” machine”outputs “ke”. A: The last letter of “learning” is “g”. The last Q:“think, machine, learning” letter of “reasoning” is “g”. The last letterof A: “think, machine” outputs ”ke”. The last “generalization” is “n”.Concatenating “g”, “g”, letter of “learning” is “g”. Concatenating “n”leads to “ggn”. So, “learning, reasoning, “ke”, “g” leads to “keg”. So,“think, generalization” outputs ”ggn”. machine, learning” outputs “keg”.

TABLE 11 Accuracy of different prompting methods with code- davinci-002on the last-letter-concatenation task with the length of listsincreasing from 4 to 12. Method L = 4 L = 6 L = 8 L = 10 L = 12 NaïvePrompting 0.0 0.0 0.0 0.0 0.0 Chain of Thought 89.4 75.0 51.8 39.8 33.6Query Recursion 94.0 88.4 83.0 76.4 74.0

Example results are also provided for the SCAN benchmark (Lake & Baroni,2018). This benchmark relates to mapping natural language commands tosequences of actions. For this example, all the prompting methods sharethe same commands, but Naïve Prompting directly maps commands to actionsequences without explanations, and Chain of Thought uses the samecommand-mapping prompts as Query Recursion, except without commandreduction. Example results are given in Table 12.

TABLE 12 Accuracies (%) of different prompting methods on the test setof SCAN under the length-based split. The results of text- davinci-002are based on a random subset of 100 commands. Method code-davinci-002code-davinci-001 text-davinci-002 Naïve 16.7 0.4 6.0 Prompting Chain of16.2 0.0 0.0 Thought Query 99.7 60.7 76.0 Recursion

Example results are also provided for the DROP benchmark. This benchmarkrelates to reading comprehension and numerical reasoning. All promptingmethods for these example results take 3 shot prompts. An example set ofprompts for Query Recursion prompting is shown in Table 13, where theprompt on the left column shows how a problem is reduced to subproblems,and the prompt on the right column shows how the subproblems aresequentially solved. Prompts for Chain of Thought here were generated bymerging Query Recursion prompts for subproblems, and prompts for NaïvePrompting were generated from the Chain of Thought prompts by removingreasoning chains. Example results are given in Table 14.

TABLE 13 Example prompts for Query Recursion Example. Example QueryBreakdown Prompt Example Query Recursion Prompt Q: The genderdistribution of the population was The gender distribution of thepopulation 50.2% male and 49.8% female. Of the adult was 50.2% male and49.8% female. Of the population, 29 people or 14.6% of the populationadult population, 29 people or 14.6% of the are between 20 and 29 yearsold. 28 people or population are between 20 and 29 years 14.1% are 30 to39, 36 people or 18.2% are 40 to old. 28 people or 14.1% are 30 to 39,36 49, and 31 people or 15.7% are 50 to 59. How people or 18.2% are 40to 49, and 31 many percent of people are not 40 to 49? people or 15.7%are 50 to 59. A: To answer the question “How many percent Q: How manypercent of people are 40 to of people are not 40 to 49?’, we need toknow 49? “How many percent of people are 40 to 49?” A: “36 people or18.2% are 40 to 49”. So the answer is 18.2%. Q: How many percent ofpeople are not 40 to 49? A: We know that 18.2% are 40 to 49. So 100% −18.2% = 81.8% are not 40 to 49. So the answer is 81.8%.

TABLE 14 Accuracies (%) of different prompting methods on the test setof SCAN under the length-based split. The results of text- davinci-002are based on a random subset of 100 commands. Non-Football (3988 cases)Football (1862 cases) Method code-davinci-002 PaLM code-davinci-002 PaLMZero-shot 43.86 48.42 51.77 44.95 Naïve 58.78 56.54 62.73 60.47Prompting Chain of 74.77 63.84 59.56 67.35 Thought Query 82.45 79.2473.42 69.98 Recursion

Example Devices and Systems

FIG. 10A depicts a block diagram of an example computing system 1 thatcan generate or implement input data structures and self-consistencyoutput sampling according to example embodiments of the presentdisclosure. The system 1 includes a computing device 2, a servercomputing system 30, and a training computing system 50 that arecommunicatively coupled over a network 70.

The computing device 2 can be any type of computing device, such as, forexample, a personal computing device (e.g., laptop or desktop), a mobilecomputing device (e.g., smartphone or tablet), a gaming console orcontroller, a wearable computing device, an embedded computing device,or any other type of computing device. In some embodiments, thecomputing device 2 can be a client computing device. The computingdevice 2 can include one or more processors 12 and a memory 14. The oneor more processors 12 can be any suitable processing device (e.g., aprocessor core, a microprocessor, an ASIC, an FPGA, a controller, amicrocontroller, etc.) and can be one processor or a plurality ofprocessors that are operatively connected. The memory 14 can include oneor more non-transitory computer-readable storage media, such as RAM,ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., andcombinations thereof. The memory 14 can store data 16 and instructions18 which are executed by the processor 12 to cause the user computingdevice 2 to perform operations (e.g., to perform operations implementinginput data structures and self-consistency output sampling according toexample embodiments of the present disclosure, etc.).

In some implementations, the user computing device 2 can store orinclude one or more machine-learned models 20. For example, themachine-learned models 20 can be or can otherwise include variousmachine-learned models such as neural networks (e.g., deep neuralnetworks) or other types of machine-learned models, including non-linearmodels or linear models. Neural networks can include feed-forward neuralnetworks, recurrent neural networks (e.g., long short-term memoryrecurrent neural networks), convolutional neural networks or other formsof neural networks. Some example machine-learned models can leverage anattention mechanism such as self-attention. For example, some examplemachine-learned models can include multi-headed self-attention models(e.g., transformer models).

In some implementations, one or more machine-learned models 20 can bereceived from the server computing system 30 over network 70, stored inthe computing device memory 14, and used or otherwise implemented by theone or more processors 12. In some implementations, the computing device2 can implement multiple parallel instances of a machine-learned model20.

Additionally, or alternatively, one or more machine-learned models 40can be included in or otherwise stored and implemented by the servercomputing system 30 that communicates with the computing device 2according to a client-server relationship.

The machine-learned models described in this specification may be usedin a variety of tasks, applications, and/or use cases.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be image data. The machine-learned model(s)can process the image data to generate an output. As an example, themachine-learned model(s) can process the image data to generate an imagerecognition output (e.g., a recognition of the image data, a latentembedding of the image data, an encoded representation of the imagedata, a hash of the image data, etc.). As another example, themachine-learned model(s) can process the image data to generate an imagesegmentation output. As another example, the machine-learned model(s)can process the image data to generate an image classification output.As another example, the machine-learned model(s) can process the imagedata to generate an image data modification output (e.g., an alterationof the image data, etc.). As another example, the machine-learnedmodel(s) can process the image data to generate an encoded image dataoutput (e.g., an encoded and/or compressed representation of the imagedata, etc.). As another example, the machine-learned model(s) canprocess the image data to generate an upscaled image data output. Asanother example, the machine-learned model(s) can process the image datato generate a prediction output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be text or natural language data. Themachine-learned model(s) can process the text or natural language datato generate an output. As an example, the machine-learned model(s) canprocess the natural language data to generate a language encodingoutput. As another example, the machine-learned model(s) can process thetext or natural language data to generate a latent text embeddingoutput. As another example, the machine-learned model(s) can process thetext or natural language data to generate a translation output. Asanother example, the machine-learned model(s) can process the text ornatural language data to generate a classification output. As anotherexample, the machine-learned model(s) can process the text or naturallanguage data to generate a textual segmentation output. As anotherexample, the machine-learned model(s) can process the text or naturallanguage data to generate a semantic intent output. As another example,the machine-learned model(s) can process the text or natural languagedata to generate an upscaled text or natural language output (e.g., textor natural language data that is higher quality than the input text ornatural language, etc.). As another example, the machine-learnedmodel(s) can process the text or natural language data to generate aprediction output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be speech data. The machine-learned model(s)can process the speech data to generate an output. As an example, themachine-learned model(s) can process the speech data to generate aspeech recognition output. As another example, the machine-learnedmodel(s) can process the speech data to generate a speech translationoutput. As another example, the machine-learned model(s) can process thespeech data to generate a latent embedding output. As another example,the machine-learned model(s) can process the speech data to generate anencoded speech output (e.g., an encoded and/or compressed representationof the speech data, etc.). As another example, the machine-learnedmodel(s) can process the speech data to generate an upscaled speechoutput (e.g., speech data that is higher quality than the input speechdata, etc.). As another example, the machine-learned model(s) canprocess the speech data to generate a textual representation output(e.g., a textual representation of the input speech data, etc.). Asanother example, the machine-learned model(s) can process the speechdata to generate a prediction output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be latent encoding data (e.g., a latent spacerepresentation of an input, etc.). The machine-learned model(s) canprocess the latent encoding data to generate an output. As an example,the machine-learned model(s) can process the latent encoding data togenerate a recognition output. As another example, the machine-learnedmodel(s) can process the latent encoding data to generate areconstruction output. As another example, the machine-learned model(s)can process the latent encoding data to generate a search output. Asanother example, the machine-learned model(s) can process the latentencoding data to generate a reclustering output. As another example, themachine-learned model(s) can process the latent encoding data togenerate a prediction output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be statistical data. Statistical data can be,represent, or otherwise include data computed and/or calculated fromsome other data source. The machine-learned model(s) can process thestatistical data to generate an output. As an example, themachine-learned model(s) can process the statistical data to generate arecognition output. As another example, the machine-learned model(s) canprocess the statistical data to generate a prediction output. As anotherexample, the machine-learned model(s) can process the statistical datato generate a classification output. As another example, themachine-learned model(s) can process the statistical data to generate asegmentation output. As another example, the machine-learned model(s)can process the statistical data to generate a visualization output. Asanother example, the machine-learned model(s) can process thestatistical data to generate a diagnostic output.

In some implementations, the input to the machine-learned model(s) ofthe present disclosure can be sensor data. The machine-learned model(s)can process the sensor data to generate an output. As an example, themachine-learned model(s) can process the sensor data to generate arecognition output. As another example, the machine-learned model(s) canprocess the sensor data to generate a prediction output. As anotherexample, the machine-learned model(s) can process the sensor data togenerate a classification output. As another example, themachine-learned model(s) can process the sensor data to generate asegmentation output. As another example, the machine-learned model(s)can process the sensor data to generate a visualization output. Asanother example, the machine-learned model(s) can process the sensordata to generate a diagnostic output. As another example, themachine-learned model(s) can process the sensor data to generate adetection output.

In some cases, the machine-learned model(s) can be configured to performa task that includes encoding input data for reliable and/or efficienttransmission or storage (and/or corresponding decoding). For example,the task may be an audio compression task. The input may include audiodata and the output may comprise compressed audio data. In anotherexample, the input includes visual data (e.g. one or more images orvideos), the output comprises compressed visual data, and the task is avisual data compression task. In another example, the task may comprisegenerating an embedding for input data (e.g. input audio or visualdata).

In some cases, the input includes visual data and the task is a computervision task. In some cases, the input includes pixel data for one ormore images and the task is an image processing task. For example, theimage processing task can be image classification, where the output is aset of scores, each score corresponding to a different object class andrepresenting the likelihood that the one or more images depict an objectbelonging to the object class. The image processing task may be objectdetection, where the image processing output identifies one or moreregions in the one or more images and, for each region, a likelihoodthat region depicts an object of interest. As another example, the imageprocessing task can be image segmentation, where the image processingoutput defines, for each pixel in the one or more images, a respectivelikelihood for each category in a predetermined set of categories. Forexample, the set of categories can be foreground and background. Asanother example, the set of categories can be object classes. As anotherexample, the image processing task can be depth estimation, where theimage processing output defines, for each pixel in the one or moreimages, a respective depth value. As another example, the imageprocessing task can be motion estimation, where the network inputincludes multiple images, and the image processing output defines, foreach pixel of one of the input images, a motion of the scene depicted atthe pixel between the images in the network input.

In some cases, the input includes audio data representing a spokenutterance and the task is a speech recognition task. The output maycomprise a text output which is mapped to the spoken utterance. In somecases, the task comprises encrypting or decrypting input data. In somecases, the task comprises a microprocessor performance task, such asbranch prediction or memory address translation.

In some embodiments, the machine-learned models 40 can be implemented bythe server computing system 40 as a portion of a web service (e.g.,remote machine-learned model hosting service, such as an onlineinterface for performing machine-learned model operations over a networkon remote servers 30). For instance, the server computing system 30 cancommunicate with the computing device 2 over a local intranet orinternet connection. For instance, the computing device 2 can be aworkstation or endpoint in communication with the server computingsystem 30, with implementation of the model 40 on the server computingsystem 30 being remotely performed and an output provided (e.g., cast,streamed, etc.) to the computing device 2. Thus, one or more models 20can be stored and implemented at the user computing device 2 or one ormore models 40 can be stored and implemented at the server computingsystem 30.

The computing device 2 can also include one or more input componentsthat receive user input. For example, a user input component can be atouch-sensitive component (e.g., a touch-sensitive display screen or atouch pad) that is sensitive to the touch of a user input object (e.g.,a finger or a stylus). The touch-sensitive component can serve toimplement a virtual keyboard. Other example user input componentsinclude a microphone, a traditional keyboard, or other means by which auser can provide user input.

The server computing system 30 can include one or more processors 32 anda memory 34. The one or more processors 32 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, anFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 34can include one or more non-transitory computer-readable storage media,such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks,etc., and combinations thereof. The memory 34 can store data 36 andinstructions 38 which are executed by the processor 32 to cause theserver computing system 30 to perform operations (e.g., to performoperations implementing input data structures and self-consistencyoutput sampling according to example embodiments of the presentdisclosure, etc.).

In some implementations, the server computing system 30 includes or isotherwise implemented by one or more server computing devices. Ininstances in which the server computing system 130 includes pluralserver computing devices, such server computing devices can operateaccording to sequential computing architectures, parallel computingarchitectures, or some combination thereof.

As described above, the server computing system 30 can store orotherwise include one or more machine-learned models 40. For example,the models 40 can be or can otherwise include various machine-learnedmodels. Example machine-learned models include neural networks or othermulti-layer non-linear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neuralnetworks, and convolutional neural networks. Some examplemachine-learned models can leverage an attention mechanism such asself-attention. For example, some example machine-learned models caninclude multi-headed self-attention models (e.g., transformer models).

The computing device 2 or the server computing system 30 can trainexample embodiments of a machine-learned model (e.g., including models20 or 40) using a pretraining pipeline (e.g., an unsupervised pipeline,a semi-supervised pipeline, etc.). In some embodiments, the computingdevice 2 or the server computing system 30 can train example embodimentsof a machine-learned model (e.g., including models 20 or 40) using apretraining pipeline by interaction with the training computing system50. In some embodiments, the training computing system 50 can becommunicatively coupled over the network 70. The training computingsystem 50 can be separate from the server computing system 30 or can bea portion of the server computing system 30.

The training computing system 50 can include one or more processors 52and a memory 54. The one or more processors 52 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, anFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 54can include one or more non-transitory computer-readable storage media,such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks,etc., and combinations thereof. The memory 54 can store data 56 andinstructions 58 which are executed by the processor 52 to cause thetraining computing system 50 to perform operations (e.g., to performoperations implementing input data structures and self-consistencyoutput sampling according to example embodiments of the presentdisclosure, etc.). In some implementations, the training computingsystem 50 includes or is otherwise implemented by one or more servercomputing devices.

The model trainer 60 can include a pretraining pipeline for trainingmachine-learned models using various objectives. Parameters of theimage-processing model(s) can be trained, in some embodiments, usingvarious training or learning techniques, such as, for example, backwardspropagation of errors. For example, an objective or loss can bebackpropagated through the pretraining pipeline(s) to update one or moreparameters of the model(s) (e.g., based on a gradient of the lossfunction). Various determinations of loss can be used, such as meansquared error, likelihood loss, cross entropy loss, hinge loss, orvarious other loss functions. Gradient descent techniques can be used toiteratively update the parameters over a number of training iterations.In some implementations, performing backwards propagation of errors caninclude performing truncated backpropagation through time. Thepretraining pipeline can perform a number of generalization techniques(e.g., weight decays, dropouts, etc.) to improve the generalizationcapability of the models being trained.

The model trainer 60 can include computer logic utilized to providedesired functionality. The model trainer 60 can be implemented inhardware, firmware, or software controlling a general-purpose processor.For example, in some implementations, the model trainer 60 includesprogram files stored on a storage device, loaded into a memory, andexecuted by one or more processors. In other implementations, the modeltrainer 60 includes one or more sets of computer-executable instructionsthat are stored in a tangible computer-readable storage medium such asRAM, hard disk, or optical or magnetic media.

The network 70 can be any type of communications network, such as alocal area network (e.g., intranet), wide area network (e.g., Internet),or some combination thereof and can include any number of wired orwireless links. In general, communication over the network 70 can becarried via any type of wired or wireless connection, using a widevariety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP),encodings or formats (e.g., HTML, XML), or protection schemes (e.g.,VPN, secure HTTP, SSL).

FIG. 10A illustrates one example computing system that can be used toimplement the present disclosure. Other computing systems can be used aswell. For example, in some implementations, the computing device 2 caninclude the model trainer 60. In some implementations, the computingdevice 2 can implement the model trainer 60 to personalize the model(s)based on device-specific data.

FIG. 10B depicts a block diagram of an example computing device 80 thatperforms according to example embodiments of the present disclosure. Thecomputing device 80 can be a user computing device or a server computingdevice. The computing device 80 can include a number of applications(e.g., applications 1 through N). Each application can contain its ownmachine learning library and machine-learned model(s). For example, eachapplication can include a machine-learned model. Example applicationsinclude a text messaging application, an email application, a dictationapplication, a virtual keyboard application, a browser application, etc.As illustrated in FIG. 10B, each application can communicate with anumber of other components of the computing device, such as, forexample, one or more sensors, a context manager, a device statecomponent, or additional components. In some implementations, eachapplication can communicate with each device component using an API(e.g., a public API). In some implementations, the API used by eachapplication is specific to that application.

FIG. 10C depicts a block diagram of an example computing device 80 thatperforms according to example embodiments of the present disclosure. Thecomputing device 80 can be a user computing device or a server computingdevice. The computing device 80 can include a number of applications(e.g., applications 1 through N). Each application is in communicationwith a central intelligence layer. Example applications include a textmessaging application, an email application, a dictation application, avirtual keyboard application, a browser application, etc. In someimplementations, each application can communicate with the centralintelligence layer (and model(s) stored therein) using an API (e.g., acommon API across all applications).

The central intelligence layer can include a number of machine-learnedmodels. For example, as illustrated in FIG. 10C, a respectivemachine-learned model can be provided for each application and managedby the central intelligence layer. In other implementations, two or moreapplications can share a single machine-learned model. For example, insome implementations, the central intelligence layer can provide asingle model for all of the applications. In some implementations, thecentral intelligence layer is included within or otherwise implementedby an operating system of the computing device 80.

The central intelligence layer can communicate with a central devicedata layer. The central device data layer can be a centralizedrepository of data for the computing device 80. As illustrated in FIG.10C, the central device data layer can communicate with a number ofother components of the computing device, such as, for example, one ormore sensors, a context manager, a device state component, or additionalcomponents. In some implementations, the central device data layer cancommunicate with each device component using an API (e.g., a privateAPI).

Example Methods

FIG. 11 depicts a flow chart diagram of an example method 1000 toperform according to example embodiments of the present disclosure.Although FIG. 11 depicts steps performed in a particular order forpurposes of illustration and discussion, the methods of the presentdisclosure are not limited to the particularly illustrated order orarrangement. The various steps of the method 1000 can be omitted,rearranged, combined, and/or adapted in various ways without deviatingfrom the scope of the present disclosure.

At 1002, a computing system can obtain an instructive sequencedescriptive of an instructive query, an instructive response, and aninstructive trace of intermediate states from the instructive query tothe instructive response. For example, illustrative instructive queries,responses, and traces are discussed with respect to FIGS. 1 to 4 . Forinstance, in some embodiments, the instructive trace can contain a chainof intermediate states or responses. For example, in some embodiments,the instructive trace can contain a chain of intermediate responses tointermediate queries (e.g., as illustrated in FIGS. 2 to 4 ).

In some embodiments, the instructive sequence can contain an input flag.For example, an instructive query can contain, for example, an inputflag signifying a start of a query (e.g., “Q:”). In some embodiments,the instructive query can also contain an output flag. For instance, anoutput flag can signify an end of a query or a beginning of a portion ofthe sequence corresponding to a response to be generated. Example flagsare shown in FIGS. 2 to 4 (e.g., “Q:”, “A:”, “Consider the followingPython function”, “[BEGIN]”, etc.).

In some embodiments, the instructive sequence can include a tokenizedrepresentation of natural language (e.g., FIGS. 2, 4 , etc.). Forinstance, the instructive sequence can be obtained by receiving anatural language sequence of words, instructions, questions,explanations, etc. and embedding the sequence into one or more tokens(e.g., word tokens, sub-word tokens, character tokens, etc.). In someembodiments, the instructive sequence can include a tokenizedrepresentation of a computer-executable coding language (e.g., FIG. 3 ).For instance, an instructive sequence can be provided to prompt themachine-learned model to simulate execution of a computer-executablescript or program (e.g., to evaluate a final output, to evaluate one ormore intermediate states of variables or parameters, etc.).

At 1004, the computing system can input to a machine-learned model, theinstructive sequence and an operative query. In some embodiments, themachine-learned model is configured to process the operative query withattention over the instructive sequence. In some embodiments, theinstructive sequence can be prepended to the operative query. Forexample, in some embodiments, the machine-learned model comprises atransformer architecture (e.g., encoder, decoder, etc.) into which theinput data structure according to the present disclosure can be input.

At 1006, the computing system can generate, using the machine-learnedmodel and responsive to the operative query, an operative response. Insome embodiments, generating the operating response can includegenerating, using the machine-learned model, a plurality of operativeresponses. In some embodiments, generating the operating response caninclude determining the operative response based on a sample of theplurality of operative responses. In some embodiments, the sample israndom. In some embodiments, the sample is based on respectiveprobabilities associated with the plurality of operative responses.

In some embodiments, determining the operative response includesdetermining a consistency metric based on the sample of the plurality ofoperative responses. For example, a consistency metric can include aself-consistency metric configured to determine internally consistentoutputs. In some embodiments, the consistency metric includes aplurality vote (e.g., a vote of output values from one or more operativeresponses). In some embodiments, the consistency metric includes amajority vote (e.g., a vote of output values from one or more operativeresponses).

In some embodiments, the method 1000 can include generating, using themachine-learned model and responsive to the operative query, anoperative trace of intermediate states from the operative query to theoperative response. In some embodiments, the vote (e.g., plurality vote,majority vote, etc.) can be based on a plurality of operative responsesrespectively associated with a plurality of diverse operative traces.

In some embodiments, the operative query can be a first query componentand the operative response can be a first response component, and themethod 1000 can include inputting, to the machine-learned model, theinstructive sequence, the first query component, the first responsecomponent, and a second query component. For instance, the method 1000can include a query recursion process flow (e.g., as described abovewith respect to FIG. 5 ).

For instance, in some embodiments, the method 1000 can includegenerating using the machine-learned model and responsive to the secondquery component, a second response component.

For instance, in some embodiments, the method 1000 can includegenerating, by the computing system and responsive to a target query,one or more query components.

For instance, in some embodiments, the method 1000 can includeinputting, to the machine-learned model, a preliminary instructivesequence including a preliminary instructive query and a preliminaryinstructive response. In some embodiments, the preliminary instructiveresponse includes a plurality of preliminary instructive querycomponents.

For instance, in some embodiments, the method 1000 can include a firstquery component and a second query component that are generated with adifferent machine-learned model other than the machine-learned modelused to obtain the first response component and the second responsecomponent.

For instance, in some embodiments, the method 1000 can include a secondquery component corresponding to the target query.

For instance, in some embodiments, the method 1000 can include, for aplurality of iterations, one or more generating and inputting operationsthat build on one another. For instance, in some embodiments, the method1000 can include, for a plurality of iterations, generating an updatedinstructive sequence based on combining one or more prior inputsequences with one or more output sequences respectively correspondingthereto; inputting, to the machine-learned model, the updatedinstructive sequence and an additional query component; and generating,using the machine-learned model and responsive to the additional querycomponent, an additional response component.

Additional Disclosure

The technology discussed herein makes reference to servers, databases,software applications, and other computer-based systems, as well asactions taken and information sent to and from such systems. Theinherent flexibility of computer-based systems allows for a greatvariety of possible configurations, combinations, and divisions of tasksand functionality between and among components. For instance, processesdiscussed herein can be implemented using a single device or componentor multiple devices or components working in combination. Databases andapplications can be implemented on a single system or distributed acrossmultiple systems. Distributed components can operate sequentially or inparallel.

While the present subject matter has been described in detail withrespect to various specific example embodiments thereof, each example isprovided by way of explanation, not limitation of the disclosure. Thoseskilled in the art, upon attaining an understanding of the foregoing,can readily produce alterations to, variations of, and equivalents tosuch embodiments. Accordingly, the subject disclosure does not precludeinclusion of such modifications, variations or additions to the presentsubject matter as would be readily apparent to one of ordinary skill inthe art. For instance, features illustrated or described as part of oneembodiment can be used with another embodiment to yield a still furtherembodiment. Thus, it is intended that the present disclosure cover suchalterations, variations, and equivalents.

Aspects of the disclosure have been described in terms of illustrativeembodiments thereof. Any and all features in the following claims can becombined or rearranged in any way possible, including combinations ofclaims not explicitly enumerated in combination together, as the exampleclaim dependencies listed herein should not be read as limiting thescope of possible combinations of features disclosed herein.Accordingly, the scope of the present disclosure is by way of examplerather than by way of limitation, and the subject disclosure does notpreclude inclusion of such modifications, variations or additions to thepresent subject matter as would be readily apparent to one of ordinaryskill in the art. Moreover, terms are described herein using lists ofexample elements joined by conjunctions such as “and,” “or,” “but,” etc.It should be understood that such conjunctions are provided forexplanatory purposes only. Clauses and other sequences of items joinedby a particular conjunction such as “or,” for example, can refer to“and/or,” “at least one of”, “any combination of” example elementslisted therein, etc. Also, terms such as “based on” should be understoodas “based at least in part on.”

What is claimed is:
 1. A computer-implemented method for improvedprompting of a machine-learned model, the method comprising: obtaining,by a computing system comprising one or more processors, an instructivesequence descriptive of an instructive query, an instructive response,and an instructive trace of intermediate states from the instructivequery to the instructive response; inputting, by the computing systemand to a machine-learned model, the instructive sequence and anoperative query, wherein the machine-learned model is configured toprocess the operative query with attention over the instructivesequence; and generating, by the computing system, using themachine-learned model and responsive to the operative query, anoperative response.
 2. The computer-implemented method of claim 1,comprising: generating, by the computing system, using themachine-learned model and responsive to the operative query, anoperative trace of intermediate states from the operative query to theoperative response.
 3. The computer-implemented method of claim 1,wherein the instructive sequence is prepended to the operative query. 4.The computer-implemented method of claim 2, wherein the instructivetrace comprises a chain of intermediate responses to intermediatequeries.
 5. The computer-implemented method of claim 1, wherein theinstructive sequence comprises an input flag and an output flag.
 6. Thecomputer-implemented method of claim 1, wherein the instructive sequencecomprises a tokenized representation of a natural language.
 7. Thecomputer-implemented method of claim 1, wherein the instructive tracecomprises one or more intermediate states of one or more variablesdeclared by a computer-executable coding language.
 8. Thecomputer-implemented method of claim 1, wherein generating the operativeresponse comprises: generating, by the computing system and using themachine-learned model, a plurality of operative responses; anddetermining, by the computing system, the operative response based on asample of the plurality of operative responses.
 9. Thecomputer-implemented method of claim 8, wherein determining theoperative response comprises: determining, by the computing system, aconsistency metric based on the sample of the plurality of operativeresponses.
 10. The computer-implemented method of claim 8, wherein thesample is based on respective probabilities associated with theplurality of operative responses.
 11. The computer-implemented method ofclaim 9, wherein the consistency metric comprises at least one of: aplurality vote, or a majority vote.
 12. The computer-implemented methodof claim 9, wherein the consistency metric comprises a vote based onoperative responses respectively associated with diverse operativetraces.
 13. The computer-implemented method of claim 1, wherein theoperative query is a first query component and the operative response isa first response component, and wherein the method comprises: inputting,by the computing system and to the machine-learned model, theinstructive sequence, the first query component, the first responsecomponent, and a second query component; and generating, by thecomputing system, using the machine-learned model and responsive to thesecond query component, a second response component.
 14. Thecomputer-implemented method of claim 13, comprising: generating, by thecomputing system and responsive to a target query, one or more querycomponents.
 15. The computer-implemented method of claim 13, comprising:inputting, by the computing system and to the machine-learned model, apreliminary instructive sequence comprising a preliminary instructivequery and a preliminary instructive response, wherein the preliminaryinstructive response comprises a plurality of preliminary instructivequery components.
 16. The computer-implemented method of claim 13,wherein the first query component and the second query component aregenerated with a different machine-learned model other than themachine-learned model used to obtain the first response component andthe second response component.
 17. The computer-implemented method ofclaim 14, wherein the second query component corresponds to the targetquery.
 18. The computer-implemented method of claim 13, comprising, fora plurality of iterations: generating, by the computing system, anupdated instructive sequence based on combining one or more prior inputsequences with one or more output sequences respectively correspondingthereto; inputting, by the computing system and to the machine-learnedmodel, the updated instructive sequence and an additional querycomponent; and generating, by the computing system, using themachine-learned model and responsive to the additional query component,an additional response component.
 19. One or more memory devices storingnon-transitory computer-readable instructions for improved prompting ofa machine-learned model, the instructions executable to cause one ormore processors to perform operations, the operations comprising:obtaining an instructive sequence descriptive of an instructive query,an instructive response, and an instructive trace of intermediate statesfrom the instructive query to the instructive response; inputting, to amachine-learned model, the instructive sequence and an operative query,wherein the machine-learned model is configured to process the operativequery with attention over the instructive sequence; and generating usingthe machine-learned model and responsive to the operative query, anoperative response.
 20. A computing system for improved prompting of amachine-learned model, the system comprising: one or more processors;and one or more memory devices storing non-transitory computer-readableinstructions that are executable to cause the one or more processors toperform operations, the operations comprising: obtaining an instructivesequence descriptive of an instructive query, an instructive response,and an instructive trace of intermediate states from the instructivequery to the instructive response; inputting, to a machine-learnedmodel, the instructive sequence and an operative query, wherein themachine-learned model is configured to process the operative query withattention over the instructive sequence; and generating using themachine-learned model and responsive to the operative query, anoperative response.